Geek asks and answers

Identification of the position in the source line from the given Boost token_iterator

If the string was processed using the Boost tokenizer, you can get the position in the original string that this iterator token points to:

boost:tokenizer<> tok( "this is the original string" );
for(tokenizer<>::iterator it=tok.begin(); it!=tok.end();++it)
{
    std::string strToken = *it;
    int charPos = it.?                /* IS THERE A METHOD? */
}

I understand that I can create a specific char_separator with a specific list of "stored delimiters" and specify keep_empty_tokens to try and track the progress of the iterator, but I was hoping there was an easier way to use only the iterator itself.

+3
source share
3 answers

This is similar to what you are looking for:

#include <string>
#include <iostream>
#include <boost/tokenizer.hpp>

int main()
{
  typedef boost::tokenizer<> tok_t;

  std::string const s = "this is the original string";
  tok_t const tok(s);
  for (tok_t::const_iterator it = tok.begin(), it_end = tok.end(); it != it_end; ++it)
  {
    std::string::difference_type const offset = it.base() - s.begin() - it->size();
    std::cout << offset << "\t::\t" << *it << '\n';
  }
}

-

+5
source

If you only need the end of the current token, the member function base() can perform this task:

std::string s = "this is the original string";
boost::tokenizer<> tok(s);
for(boost::tokenizer<>::iterator it=tok.begin(); it!=tok.end();++it)
{
    int charPos = it.base() - s.begin();
}

, , boost::tokenizer.

+1

What about:

 int charPos = it - tok.begin() ;
0
source

More articles:


All Articles