Big Data Engineer

Location: Herndon, VA, United States
Date Posted: 05-20-2016
Our client is utilizing cutting edge open source technologies and frameworks to derive actionable insights from massive amounts of data. Our solution provides capability to ingest petabytes of structured and unstructured data and build analytic pipelines that crunch through this data. 

We are looking for a Big Data Engineer that will utilize our base platform to develop ingestion and analytic pipelines utilizing a multitude of open source technologies. The primary focus will be on implementing optimal solutions utilizing best practices that can be shared across all our clients. This position requires working with technical leads, data scientists and project managers in a Scrum based Agile environment. 

  • Create, configure, implement, document and maintain ingestion, enrichment and analytic pipelines using distributed big data platform
  • Extend base platform functionality by adding new ingestion and analytic sources
  • Identify, evaluate and implement big data tools and frameworks required to provide requested capabilities

  • Java development experience
  • Scripting language experience (Perl, Python, JavaScript)
  • Proficient understanding of distributed computing principles
  • Proficiency with Hadoop v2, MapReduce, HDFS
  • Experience with building stream-processing systems, using solutions such as Storm or Spark 
  • Experience with integration of data from multiple data sources
  • Experience with NoSQL databases, such as Elasticsearch, MongoDB, Cassandra
  • Knowledge of various ETL techniques and frameworks, such as Flume, Logstash
  • Experience with various messaging systems, such as Kafka or RabbitMQ
  • Experience with Cloudera/Hortonworks distributions
  • Bachelor's degree or equivalent professional experience
  • Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
  • Management of Hadoop cluster, with all included services 
  • Ability to solve any ongoing issues with operating the cluster 
this job portal is powered by CATS