Task-oriented dialog agents converse with a user towards the goal of accomplishing a specific task and often interact with a knowledge-base (KB). For example, a restaurant reservation agent will be grounded to a KB that contains the names of restaurants, and their details. To train a TOD system in a end-to-end manner, the system requires two annotations on the chat logs: (1) where in the dialog the KB was accessed and (2) what was the KB query used. As such KB annotations are not present in real-world dialog, a domain expert annotates them. We propose a framework and approach which can learn a TOD systems without the need for such annotations. We show that the proposed approach attains performance comparable to approaches that are trained using these annotations.