The Trident or Storm Topology That Writes About Redis

I have a problem with the topology. I am trying to explain the workflow ... I have a source that emits ~ 500k tuples every 2 minutes, these tuples should be read by the nose and processed exatly once as one object (I think the packet is in the trident). After that, bolt / function / what else? ... should add a timestamp and save the tuples in Redis.

I tried to implement a Trident topology with a function that saves all tuples in Redis using the Jedis object (Redis library for Java) in this Function class, but when I deploy, I get a NotSerializable Exception on this object.

My question is . How can I implement a function that writes this batch of tuples to Redis? Reading on the Internet I cannot find any example that writes from a function in Redis or any example using a State object in Trident (maybe I should use it ...)

My simple topology is:

TridentTopology topology = new TridentTopology();
topology.newStream("myStream", new mySpout()).each(new Fields("field1", "field2"), new myFunction("redis_ip", "6379"));

Thanks in advance

+3
source share
2 answers

(a response to the state in general, as the specific issue related to Redis seems to be resolved in other comments)

Storm , , Storm ( "" ) ( Stort "spouts" ), , ( "" ) ( "" ). - , " ": , Storm , node ,

Storm, - : partitionPersist persistentAggregate, node DAO, - "" (.. , , ), ( "" ) / - , HashMap, (Storm DB "MapState" "", ).

, , , Storm , : , / , , , . , Storm, , , , . , Storm (OpaqueMap), , . , [1,2,3,4,5], , , 15, "" - . OpaqueMap , . , , Storm , , .

, 2 .

http://svendvanderveken.wordpress.com/2013/07/30/scalable-real-time-state-update-with-storm/

http://svendvanderveken.wordpress.com/2014/02/05/error-handling-in-storm-trident-topologies/

: , Storm - : , (, Kafka 0MQ), Storm . , , : " ", , , , .

+7
+1

All Articles