Arabic fluency is only one part of the decision
An Arabic model can produce an elegant answer while selecting the wrong product, omitting a delivery condition or misreading the quantity in a scanned document. A business evaluation therefore needs to score the whole task. The question is not only whether the answer reads well, but whether the system reached a correct, permitted and recoverable commercial state.
Begin with the work you want to support. An Arabic knowledge assistant needs grounded retrieval and faithful citations. A quotation workflow needs accurate entities, units and prices. A voice enquiry workflow also depends on transcription. Keep those capabilities separate in the scorecard so that a failure in one component is not hidden by a fluent final response.
Test whether required fields are extracted accurately from the source.
Check how the system handles conflicting evidence and uncertainty.
Use rules, verified calculations and permissions to control the final action.
Read the visual explanation
A practical sequence for this decision. Apply your actual records, responsibilities and acceptance criteria.
What Jais and Falcon Arabic bring to a shortlist
Jais is an Arabic-centric model family with published model cards and research. Falcon Arabic is a TII model whose release describes training on native Arabic material, including Modern Standard Arabic and regional dialects. Both are relevant candidates for testing. Their provenance and intended language coverage do not establish that a particular version is suitable for your industry, licence requirements or deployment environment. [1] [2] [3]
Compare them with multilingual models your organization is already permitted to use. Record the exact model version, prompt, retrieval configuration, quantization if applicable and inference environment. Do not compare a tuned Arabic model with an untuned alternative using different context and then present the result as a universal ranking.
Check the model card and licence of the specific artifact you plan to deploy. Open weights, commercial permission, API availability and a supported managed service are different things. Self-hosting adds capacity planning, patching, monitoring and incident response. A downloadable model is not a completed production service.
Build a test set that exposes operational failure
Sample the actual language mix: formal Arabic, the dialects your buyers use, English, Arabic-English code-switching and romanized terms. Use permissioned or appropriately de-identified records. Keep a separate holdout set that is not used while refining prompts. Include common requests as well as the long tail that causes the most costly mistakes.
For a distributor, one case might ask for twelve cartons of a SKU with an English name and Arabic delivery instructions. Another might refer to the previous order without identifying the item. A third might request a substitute whose specification is almost, but not quite, compatible. The expected answer should identify what is known, what source proves it and what requires clarification.
Include adversarial document content that tries to override business rules. A supplier PDF is evidence to read, not an authority to instruct the agent to reveal records or approve a payment. Test whether the workflow keeps source material separate from the instructions governing its actions.
Add a no-answer condition. The right behavior may be to ask for the missing unit, flag a contradiction or route a question to a specialist. If every test is scored on producing an answer, the evaluation rewards confidence when it should reward restraint.
Score six dimensions independently
Measure entity and number accuracy: customer, product code, amount, currency, date and unit. Then measure retrieval: did the system find the current approved record rather than a plausible but outdated one? Score answer faithfulness separately: can each important statement be traced to evidence?
Next score action correctness. Did the system choose the permitted tool, supply validated arguments and avoid duplicate writes? Score escalation quality: did it identify the precise missing fact and route it to the right owner? Finally score language usability with bilingual reviewers who understand the industry.
Do not collapse these dimensions too early. A weighted average can conceal a critical error. Define hard failures such as changing a SKU, inventing availability, exposing another account’s data or committing an unauthorized price. These should block release even if the language score is high.
RAG and terminology management often matter as much as model choice
Retrieval-augmented generation supplies relevant records to a model at the time of the request. In bilingual work, retrieval can fail before generation begins: the buyer uses an Arabic phrase, while the product master uses an English technical label. Maintain reviewed aliases and preserve canonical identifiers. Evaluate retrieval across languages, not just within one language.
Keep the source document and its effective date attached to the retrieved fact. Separate factual records from examples and marketing copy. A previous quote may explain a past decision, but it is not automatically the price for a new order. The workflow must know which source governs the current action.
Fine-tuning may help a recurring task, but it should not be the default answer to missing business context. First identify whether the error is due to retrieval, ambiguous data, instructions, tool behavior or language capability. Each has a different remedy, and a larger model does not fix all of them.
Cost the completed task and keep the human loop visible
Measure latency and total cost over the complete sequence, including retries, retrieval, model calls, infrastructure and specialist correction. Compare cost per accepted case at a fixed quality threshold. Cheap tokens can become expensive when the workflow repeatedly hands incomplete work to a person.
Release in stages: shadow evaluation, supervised drafting, then limited actions with explicit approvals. Keep a decision log and rerun the holdout tests when the model, prompt or source system changes. Monitor quality by language and request type rather than only an overall success rate.
A useful human handoff contains the original request, retrieved evidence, uncertainty and proposed next action. That is the point where Looqus specialists can resolve blockers and see the work through. This article presents an evaluation method, not an independently measured leaderboard or a claim that Looqus has benchmarked every model listed.
Put the decisions side by side
| Evaluation dimension | What to inspect | Acceptance decision |
|---|---|---|
| Exact facts | SKU, quantity, currency, date | Hard fail for altered identifiers |
| Grounding | Correct current source | No unsupported availability or price |
| Action | Valid tool and permitted arguments | No unauthorized or duplicate write |
| Handoff | Specific uncertainty and responsible owner | Clarification rather than invented answer |
| Language | Dialect and terminology usability | Domain-aware bilingual review |
| Economics | Cost per accepted case | Include retries and human correction |
Sources and scope
Worked scenarios are illustrative designs, not customer results or performance guarantees. Confirm current product availability and applicable requirements before implementation.
AI does the work.We own the outcome.
Test Arabic requests against the actual business task, with clear checks for meaning, order accuracy and the corrections needed before completion.
- Agree the result
- Connect the work
- Own the follow-through
Follow the work, one decision at a time.
- Representative Arabic inputs
- Verified retrieval
- Constrained action
- Bilingual quality review