Skip to content

Conversation

@LLParse
Copy link

@LLParse LLParse commented Jul 15, 2015

  • Rework configuration system to use typesafe snakeyaml api
  • Minimize dependency scope, remove unused dependencies for ~50% size savings in built artifacts
  • Define protocol buffer messages for journal, name, data nodes
  • Remove configuration logic from artifact server, use Mesos data fields to pass protobuf between scheduler & executor (this fixes idempotency [WIP, name/data nodes not quite finished])
  • Refactor/simplify executor system, emit hdfs-site.xml using {{mustache}} template and task protobuf data
  • Pool enough offers to launch journalnodes all at once (TODO: save journal rpc addresses in persistent state, use ports in offer when configured ports are set to '0', genericize pooled offer logic to work for name/data node acquisition phases, wrap offer system in timer to prevent holding onto resources indefinitely in too-few-resources scenarios)
  • TODO: major scheduler refactor, tweak name/datanode config

@LLParse LLParse changed the title accept offers based on required journalnode count, populate config info in mesos TaskInfo Refactor Jul 21, 2015
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using a yaml config here, what I would prefer to do is something similar to the following PR (which didn't get finished), mesosphere-backup#123. I would like to use environment variables or command line arguments (similar to Cassandra). These values could then be used to generate the *-site.xml configs used by HDFS.

@elingg
Copy link

elingg commented Aug 3, 2015

Overall comment, this would need to be rebased with mesosphere/master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants