Ordered receive in iterator within iterator

Sep 21, 2011 at 11:31 PM
Edited Sep 21, 2011 at 11:31 PM

Hi Thomas,

Long time no speak - Hope you are well!

Do you have any ideas on the following scenario. I have an Iterator such like,

[XcoConcurrent]

IEnumerator<ITask> ProcessEntriesForMediaQueue(QueueToProcess qtp)

In this iterator I am constructing what is effectively an instruction set in XML format that is handed off to another process once all instructions are gathered. To get each XML instruction, i need to go off to the database and retrieve a PDF, write it to disk and then run some logic to contruct the individual XML instruction so each one of these takes some time. I would like to put this logic in another iterator however I need all my returned calls to be in order as each individual XML instruction must be added back to the main XML in the order it was called.

i.e. PseudoCode(ish) it looks something like 

IEnumerator<ITask> ProcessEntriesForMediaQueue(QueueToProcess qtp)

{

XmlTextWriter instructionSet.......

foreach(var item in qtp.ItemsToProcess)

{

yield return new IterativeTask<XmlInstruction>(item, ChildIteratorMethod);

// Get result from response port

instructionSet.WriteXml(result.xml); // These must be added in the order they are dispatched

}

}

IEnumerator<ITask> ChildIteratorMethod(XmlInstruction instruct)
{
//Do stuff here to grab PDF from database, logic in what the instruction should be, return XML of instruction

 

Can AppSpace help with this or can you provide a simple example of how to do this?

Many thanks in advance for your help

Mike

Sep 22, 2011 at 3:08 PM

Hi Mike,

Yes it is possible to have nested iterators, which can come in really handy when your processing method gets more complex and you don't want to put all the logic into a single method. To do this, use IEnumerable<ITask> (not IEnumerator) as return for your nested iterator method, then you can just iterate and yield over your nested method. Like this:

[XcoConcurrent]
IEnumerator<ITask> Process(QueueToProcess qtp)
{
  foreach(var item in qtp.ItemsToProcess)
  {
    yield return something; // yield some tasks directly if needed...

   foreach (var innerTask in ChildIteratorMethod(item))
     yield return innerTask; //yields the tasks from the nested iterator method...

  }
}

IEnumerable<ITask> ChildIteratorMethod(...) { ... }


Hope this helps!

Best regards
Thomas

Sep 22, 2011 at 3:26 PM

Hi Thomas,

Thanks for the reply.

In your example does each call in the foreach to the ChildIteratorMethod get scheduled by the CCR so that the foreach does not have to wait for the return? If this is right, when eventually the method does return, it it in order that the foreach was called so that the result of ChildIteratorMethod I can safely add to my XML instruction set (declared before the foreach)  knowing it will be in the order that I called each method?

Thanks in advance for the help

Kind regards

Mike

Sep 22, 2011 at 3:56 PM

Hi mike,

The Iterator will always continue only when the previous task is fulfilled and by that the ccr hands back the control to your method. So there won't be any tasks running in parallel (at least not within a single worker method call), and the correct order will be guaranteed.

Best regards
Thomas

Sep 22, 2011 at 4:48 PM

Hi Thomas,

Thanks for this.

So how would I make it that the ChildMethod calls run in parallel but are returned in order because each call takes a few seconds and I would like x number of these to run in parallel to speed things up?

Kind regards

Mike

Sep 23, 2011 at 12:46 PM

Hi mike,

This cannot be done automatically - you'll need an additional processing step that aggregates the results, something like:

  1. Post the child tasks to a concurrently running worker or port, so that they are processed in parallel
  2. Have a response port that collects the responses and puts them in the needed order
  3. After collection of results is complete, the list of results can be handed back to your iterator method via a second response port

Best regards
Thomas