Patch for getting a sample of in-memory#1
Open
Sanil15 wants to merge 21 commits into
Open
Conversation
This patch adds the capability to register arbitrary timers for both high-level and low-level api. For high-level, InitableFunction will pass the TimerRegistry to user through the new OpContext, and user will implement the TimerFunction to get timer notifications. For low-level api, user can register timer in the TaskContext, and then implements the TimerCallback for specific timer actions. Author: xiliu <xiliu@linkedin.com> Author: xinyuiscool <xinyuliu.us@gmail.com> Author: xinyuiscool <xiliu@linkedin.com> Reviewers: Pateek M <prateekm@gmail.com> Closes apache#419 from xinyuiscool/SAMZA-1498
Tested with ./gradlew clean build, which is passing Author: sanil15 <sanil.jain15@gmail.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#439 from Sanil15/SAMZA-1602
Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#437 from shanthoosh/fix_zkutils_get_processor_data
…riteCheckpoint Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com> Reviewers: Jagadish <jagadish@apache.org> Closes apache#438 from shanthoosh/SAMZA-1589
Right now when we try to use the SamzaSqlApplicationRunner in yarn mode it goes into the infinite loop because the appRunnerConfig that we try to set is being overwritten by the appRunnerConfig passed by the job and SamzaSqlApplicationRunner keeps creating it again and again in an infinite loop. Fix : Userconfig for app.runner.class should not override the computed one. Added a test case to validate this. Author: Srinivasulu Punuru <spunuru@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#440 from srinipunuru/bug-fix.1
Please see commit messages for detailed descriptions. Author: Peng Du <pdu@linkedin.com> Reviewers: Jagadish <jagadish@apache.org>, Wei Song <wsong@linkedin.com> Closes apache#432 from pdu-mn1/remote-table-0222
fix HdfsFileSystemAdapter to get files recursively Author: Hai Lu <halu@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#447 from lhaiesp/master
…tor API to compare the offsets Author: Aditya Toomula <atoomula@linkedin.com> Reviewers: Jagadish <jagadish@apache.org> Closes apache#443 from atoomula/bootstrap
bharathkk
force-pushed
the
single-node-testing
branch
from
March 15, 2018 21:16
cbc3dce to
e9eaa63
Compare
Two issues I found during testing: 1) medaDataCache.getSystemStreamMetadata(): if we pass in partitionOnly to be true, it will actually not store the metadata into the cache, resulting every call being another query to kafka. I turned off the partitionOnly in order to make it in the cache. 2) change the log for info to debug. Author: xinyuiscool <xiliu@linkedin.com> Reviewers: Boris S <sborya@gmail.com> Closes apache#444 from xinyuiscool/SAMZA-1615
avro writer in HDFS system producer to support generic record Author: Hai Lu <halu@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#449 from lhaiesp/master
…StreamAppender due to bug. Due to a intermittent bug that causes the explicit stream creation in `StreamAppender` to hang, a hidden config is added to enable/disable explicit stream creation. By default this is disabled, which reverts to the previous behavior. When the intermittent hang bug is fixed, the config will either be removed or made public. Author: Daniel Nishimura <dnishimura@gmail.com> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#442 from dnishimura/samza-1608-disable-streamappender-create-stream
AvroDataFileHdfsWriter should include avro as the file suffix as some pig jobs couldn't read the avro files if they don't come with the proper suffix Author: Hai Lu <halu@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#452 from lhaiesp/master
Author: Aditya Toomula <atoomula@linkedin.com> Reviewers: Yi Pan <nickpan47@gmail.com> Closes apache#425 from atoomula/join
Fixes for Bugs - SAMZA-1571 Make Eventhubs system configs compatible with Samza standalone. - SAMZA-1624 EventHub system should prefix the configs with senstive for SasKey and SasToken - SAMZA-1625 EventHub systemAdmin is swallowing exceptions - SAMZA-1626 EventHub system admin is not returning the metadata for all the ssps requested for Description 1. Right now event hub doesn't follow the samza's config convention of naming the secrets as "sensitive" so that they are masked before they are logged. 2. Event hub configs uses the old system.<systemName>.streams.<streamName> which is blacklisted in Samza standalone. So moving these configs to newer <streams>.<streamid> 3. Wrapping the underlying exception properly in the SamzaException in EventHubSystemAdmin 4. Porting Bharat's fix to return the metadata for all the ssps requested for in EventHubSystemAdmin Author: Srinivasulu Punuru <spunuru@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#453 from srinipunuru/eh.1
…unhandled Errors If an Action executed in the scheduler throws an Error (or other Throwable?) besides Exception, it is silently lost since the Action/Runnable wrapper only catches Exception, not Throwable. This made my troubleshooting of an issue very difficult. Made it seem like the code was "hung" when really it had thrown a "NoSuchMethodError" (Error instead of Exception) due to a simple dependency issue on my side. Catching Throwable instead ensures this is handled and propagated properly. Author: thunderstumpges <tstumpges@ntent.com> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#450 from thunderstumpges/scheduler-catch-throwable
EventHubs have restriction on maximum message sizes that can be allowed. Adding a `systems.%s.eventhubs.skipMessagesLargerThanBytes` that can be used in the event hub system to make it skip messages that are larger than specific bytes so that we don't even try to send those large messages because EventHubs will reject them anyways. Author: Srinivasulu Punuru <spunuru@linkedin.com> Reviewers: Xinyu Liu <xinyuliu.us@gmail.com> Closes apache#454 from srinipunuru/skip.2
Currently each upstream task needs to broadcast to every single partition of intermediate streams in order to aggregate watermarks in the consumers. A better way to do this is to have only one downstream consumer doing the aggregation, and then broadcast to all the partitions. This is safe as we can prove the broadcast watermark message is after all the upstream tasks finished producing the events whose event time are before the watermark. This reduced the full message count from O(n^2) to O(n). Author: xinyuiscool <xiliu@linkedin.com> Reviewers: Boris S <sborya@gmail.com> Closes apache#456 from xinyuiscool/SAMZA-1627
bharathkk
force-pushed
the
single-node-testing
branch
from
March 28, 2018 19:00
f79aee5 to
0fa31f3
Compare
bharathkk
force-pushed
the
single-node-testing
branch
from
March 28, 2018 19:10
0fa31f3 to
3580632
Compare
bharathkk
force-pushed
the
single-node-testing
branch
from
March 29, 2018 00:34
93177e0 to
3b40689
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.