Your task Suppose you are organizing a shared task at an information retrieval conference, as part of the job you need to create a corpus using the pooling method. One of the search queries you are dealing with is “Graduation information at Colchester campus” the background of this query is “The user is coming to the Colchester campus to attend his graduation with his parents, they travel by car. And want to find some relevant information about this trip.” The following documents are retrieved by three search engines (You need to click each link and check the contents of the webpage) https://www.essex.ac.uk/graduation/about-the-ceremony https://www1.essex.ac.uk/site/a-z.aspx https://www.essex.ac.uk/alumni/honorary/honorary-graduates https://www.essex.ac.uk/disclaimer/accessibility-hub/www1-applications https://www.essex.ac.uk/life/southend-campus/local-area https://www.essex.ac.uk/research/areas https://www.essex.ac.uk/graduation/schedules https://www.essex.ac.uk/graduation/tickets-and-information https://www.essex.ac.uk/alumni/awards/alumnus-of-the-year/2008 https://www.essex.ac.uk/student/travel-and-transport/parking-for-students https://www.essex.ac.uk/life/colchester-campus/how-to-get-here https://www.essex.ac.uk/about/how-we-teach https://www.essex.ac.uk/arena/how-to-get-here https://www.essex.ac.uk/life/southend-campus https://www.essex.ac.uk/visit-us/open-days As you would expect, the systems ranked the documents in different orders, more specifically the documents are ranked by the three IR systems (IR1, IR2 and IR3) in the following order: IR1 7 15 13 9 8 1 5 2 14 4 10 3 11 6 12 IR2 8 9 1 3 4 2 14 7 13 5 11 10 15 6 12 IR3 6 1 7 4 3 14 8 5 9 12 13 15 2 11 10 Pooling (35%) The first task is to make the relevant judgements. You are using the pooling method with a K=5. i.e. only consider the top 5 results returned by the three IR systems, remember documents outside the pool are automatically considered to be irrelevant P/R@5 (15%) Once you have your relevance labels from pooling, you can explore the effect of each IR system on the evaluation results. In this task, you will need to compute the P@5 and R@5 for all three systems. Average Precision (15%) Next let’s compute the average precision for all three systems. Again, we will use the relevance labels from pooling. Discounted Cumulative Gain (15%) Next you need to compute the DCG for all systems, here we use the binary judgement, i.e., rel = 0/1 for non-relevant and relevant documents respectively the label is decided by the pooling step. Could you please answer the question in this format Pooling Include here your judgments, the reason you made your decision and the details of how you did the pooling. Doc Judgment Reason 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. P/R@5 For each system, compute P/R@5, please include details of the computation. Systems P@5 R@5 IR1: IR2: IR3: Average Precision For each system, compute average precision, please include details of the computation. Systems Average Precision IR1: IR2: IR3: Discounted Cumulative Gain For each system, compute discounted cumulative gain, please include details of the computation. Systems Discounted Cumulative Gain IR1: IR2: IR3: Instructions: Complete all the parts. Show all the calculations. Do not use AI.

Solved: Your task Suppose you are organizing a shared task at an information r

Home
questions and answers
your task suppose you are organizing a shared task at an information r

Question

Your task Suppose you are organizing a shared task at an information retrieval conference, as part of the job you need to create a corpus using the pooling method. One of the search queries you are dealing with is “Graduation information at Colchester campus” the background of this query is “The user is coming to the C