Geek asks and answers

Using Boost Tokenizer escaped_list_separator with various options

Hello, I'm trying to get the tokenizer to work using the tokenizer boost class. I found this tutorial in the documentation on speeding up:

http://www.boost.org/doc/libs/1 _36 _0 / libs / tokenizer / escaped _list _separator.htm

the problem is that I cannot get an argument for the escape-_list _ separator ("," "," ");

but if I modify the boost / tokenizer.hpp file, it works. but this is not an ideal solution, I wondered if there is something that I do not see in order to get different arguments in escape-_list_separator.

I want it to be split into spaces with the "and" character for escaping and without an escape character inside the specified string.

this is used for the argument analysis system in the game console.


include <iostream>
include <boost/tokenizer.hpp>
include <string>

int main() { using namespace std; using namespace boost; string s = "exec script1 \"script argument number one\""; string separator1("");//dont let quoted arguments escape themselves string separator2(" ");//split on spaces string separator3("\"\'");//let it have quoted arguments tokenizer<escaped_list_separator<char>(separator1,separator2,separator3)> tok(s); for(tokenizer<escaped_list_separator<char>(separator1,separator2,separator3)>::iterator beg=tok.begin(); beg!=tok.end();++beg) { cout << *beg << "\n"; } }

the error from visual studio 2005 is error C2974: 'boost :: tokenizer': invalid template argument for 'TokenizerFunc', type expected

EDIT: This question was called by Ferruchio and explained to everyone who thanked everyone.

+4
source share
3 answers

try the following:

#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>

int main()
{
    using namespace std;
    using namespace boost;
    string s = "exec script1 \"script argument number one\"";
    string separator1("");//dont let quoted arguments escape themselves
    string separator2(" ");//split on spaces
    string separator3("\"\'");//let it have quoted arguments

    escaped_list_separator<char> els(separator1,separator2,separator3);
    tokenizer<escaped_list_separator<char>> tok(s, els);

    for(tokenizer<escaped_list_separator<char>>::iterator beg=tok.begin(); beg!=tok.end();++beg)
    {
        cout << *beg << "\n";
    }
}
+14
source

It looks like you are declaring your tokenizer incorrectly.

typedef boost::tokenizer< boost::escaped_list_separator<char> > Tokenizer;
boost::escaped_list_separator<char> Separator( '\\', ' ', '\"' );
Tokenizer tok( s, Separator );

for( Tokenizer::iterator iter = tok.begin(); iter != tok.end(); ++iter )
{ cout << *iter << "\n"; }

You want to make a boost::tokenizer< boost::escaped_list_separator< char > >typed object with boost::escaped_list_separator< char >a separator object as its TokenizerFunc.

+4
source

Another important point is that if the user wants to display a double quote as a result, the built-in quote (") described on Wikipedia ( as here ) should be replaced with a line (\\\")where it \\means escape char and \"means quote mark. Thus, the quotation mark will be displayed in the output, for example, the following code snippet.

typedef boost::escaped_list_separator<char> std_token;
typedef boost::tokenizer<boost::escaped_list_separator<char>> tokenizer;

std_token a_token( "\\", ",", "\"" );
std::string s = "\"Boost,\\\" C++ Libraries\" ";
tokenizer tok{ s, a_token };
for ( const auto &t : tok )
    std::cout << t << '\n';

This is a typical result for this.

0
source

More articles:


All Articles