Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion antora.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ asciidoc:
pulsar-heartbeat-repo: 'https://github.com/datastax/pulsar-heartbeat'
pulsar-sink-repo: 'https://github.com/datastax/pulsar-sink'
pulsar-ansible-repo: 'https://github.com/datastax/pulsar-ansible'
pulsar-helm-chart-repo: 'https://github.com/datastax/pulsar-helm-chart'
pulsar-helm-chart-repo: 'https://github.com/datastax/pulsar-helm-chart' # deprecated, use kaap-operator-repo, kaap, and kaap-short instead
pulsar-openid-connect-repo: 'https://github.com/datastax/pulsar-openid-connect-plugin'
pulsar-repo: 'https://github.com/datastax/pulsar'
pulsar-beam-repo: 'https://github.com/kafkaesque-io/pulsar-beam'
Expand Down
3 changes: 0 additions & 3 deletions local-preview-playbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -221,9 +221,6 @@ asciidoc:
cr-short: 'CR'
crd: 'custom resource definition (CRD)'
crd-short: 'CRD'
# Custom attributes only used in ragstack-ai
astra_db: 'Astra DB'
astra_ui: 'Astra Portal'
# Antora Atlas
primary-site-url: https://docs.datastax.com/en
primary-site-manifest-url: https://docs.datastax.com/en/site-manifest.json
Expand Down
17 changes: 9 additions & 8 deletions modules/ROOT/pages/faqs.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,13 @@ They include Minikube, K8d, Kind, Google Kubernetes Engine (GKE), Microsoft Azur

There are several public repos, each with a different purpose. See:

* {pulsar-repo}[{pulsar-repo}] : This is the distro repo (a fork of apache/pulsar).
* {pulsar-admin-console-repo}[{pulsar-admin-console-repo}] : This is the repo for the {pulsar-short} admin console, which allows for the configuration and monitoring of {pulsar-short}.
* {pulsar-repo}[{pulsar-repo}]: This is the distro repo (a fork of `apache/pulsar`).
* {pulsar-admin-console-repo}[{pulsar-admin-console-repo}]: This is the repo for the {pulsar-short} admin console, which allows for the configuration and monitoring of {pulsar-short}.
* {pulsar-heartbeat-repo}[{pulsar-heartbeat-repo}] : This is a monitoring/observability tool for {pulsar-short} that tracks the health of the cluster and can generate alerts in Slack and OpsGenie.
* {pulsar-helm-chart-repo}[{pulsar-helm-chart-repo}] : This is the Helm chart for deploying the {company} {pulsar-short} Distro in an existing Kubernetes cluster.
* {pulsar-sink-repo}[{pulsar-sink-repo}] : This is the {company} {pulsar} Connector (`pulsar-sink`) repo.
* https://github.com/datastax/burnell[https://github.com/datastax/burnell] : This is a utility for {pulsar-short} that provides various functions, such as key initialization for authentication, and JWT token creation API.
* {kaap-operator-repo}[{kaap-operator-repo}]: Use the {kaap-short} to deploy the {company} {pulsar-short} Distro in an existing Kubernetes cluster.
+
{company} recommends the {kaap-short} as a replacement for the deprecated {pulsar-helm-chart-repo}[{company} {pulsar-short} Helm chart].
* {pulsar-sink-repo}[{pulsar-sink-repo}]: This is the {company} {pulsar} Connector (`pulsar-sink`) repo.

== Is there a prerequisite version of Java needed for the {product} installation?

Expand All @@ -57,17 +58,17 @@ The {product} distribution is designed for Java 17. However, because the product
== What are the install options for {product}?

* **{kaap-short} Helm chart (Recommended)**: Use the {company} {kaap-operator-repo}[{kaap-short} Helm chart], which provides Kubernetes-native autoscaling and simplified management for {pulsar} clusters. For more information, see the xref:kaap-operator::index.adoc[{kaap}] documentation.
* **{product} Helm chart**: Use the Helm chart provided at {pulsar-helm-chart-repo}[{pulsar-helm-chart-repo}] to install {product} in an existing Kubernetes cluster on your laptop or hosted by a cloud provider.
* **{company} {pulsar-short} Helm chart (Deprecated)**: The Helm chart provided at {pulsar-helm-chart-repo}[{pulsar-helm-chart-repo}] is deprecated and no longer maintained. Use the {kaap-short} instead.
* **Tarball**: Use the tarball provided at {pulsar-repo}/releases[{pulsar-repo}/releases] to install {product} on a server or VM.
* **Ansible**: Use the {company} Ansible scripts provided at {pulsar-ansible-repo}[{pulsar-ansible-repo}] to install {product} on a server or VM with our provided playbooks.

== How do I install {product} in my Kubernetes cluster?

Follow the full instructions in xref:install-upgrade:quickstart-helm-installs.adoc[Quick Start for Helm Chart installs].
Follow the full instructions in xref:install-upgrade:quickstart-helm-installs.adoc[].

== How do I install {product} on my server or VM?

Follow the full instructions in xref:install-upgrade:quickstart-server-installs.adoc[Quick Start for Server/VM installs].
Follow the full instructions in xref:install-upgrade:quickstart-server-installs.adoc[].

== What task can I perform in the {product} Admin Console?

Expand Down
71 changes: 71 additions & 0 deletions modules/components/attachments/beam/values.yaml
Comment thread
aimurphy marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
extra:
broker: false
brokerSts: true
pulsarBeam: true
proxy: true
wsproxy: false
bastion: false

zookeeper:
replicaCount: 1

bookkeeper:
replicaCount: 1

broker:
component: "broker" # this has to match brokerSts naming
replicaCount: 0

brokerSts:
component: "broker" # this has to match broker naming
replicaCount: 1
functionsWorkerEnabled: true # to overcome the need for functions working
ledger: # the default assumes 2 bookies but this chart only deploys 1
defaultEnsembleSize: 1
defaultAckQuorum: 1
defaultWriteQuorum: 1

# this is a legacy value that the beam sub-chart uses
tokenServer:
allowedRoles: superuser,admin,websocket,proxy

proxy:
component: "proxy"
replicaCount: 1
service:
ports:
- name: "pulsarbeam"
port: 8085
protocol: TCP
- name: "http"
port: 8080
protocol: TCP
- name: "pulsar"
port: 6650
protocol: TCP

pulsarBeam:
component: "pulsarbeam"
replicaCount: 1

image:
proxy:
repository: datastax/lunastreaming
pullPolicy: IfNotPresent
tag: 2.10_2.4
brokerSts:
repository: datastax/lunastreaming
pullPolicy: IfNotPresent
tag: 2.10_2.4
zookeeper:
repository: datastax/lunastreaming
pullPolicy: IfNotPresent
tag: 2.10_2.4
bookkeeper:
repository: datastax/lunastreaming
pullPolicy: IfNotPresent
tag: 2.10_2.4
pulsarBeam:
repository: kesque/pulsar-beam
pullPolicy: IfNotPresent
tag: "1.0.0"
50 changes: 50 additions & 0 deletions modules/components/attachments/client.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Configuration for pulsar-client and pulsar-admin CLI tools

# URL for Pulsar REST API (for admin operations)
# For TLS:
# webServiceUrl=https://localhost:8443/
webServiceUrl=http://127.0.0.1:8080/

# URL for Pulsar Binary Protocol (for produce and consume operations)
# For TLS:
# brokerServiceUrl=pulsar+ssl://localhost:6651/
brokerServiceUrl=pulsar://127.0.0.1:6650/

# Authentication plugin to authenticate with servers
# e.g. for TLS
# authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
authPlugin=

# Parameters passed to authentication plugin.
# A comma separated list of key:value pairs.
# Keys depend on the configured authPlugin.
# e.g. for TLS
# authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem
authParams=

# Allow TLS connections to servers whose certificate cannot be
# be verified to have been signed by a trusted certificate
# authority.
tlsAllowInsecureConnection=false

# Whether server hostname must match the common name of the certificate
# the server is using.
tlsEnableHostnameVerification=false

# Path for the trusted TLS certificate file.
# This cert is used to verify that any cert presented by a server
# is signed by a certificate authority. If this verification
# fails, then the cert is untrusted and the connection is dropped.
tlsTrustCertsFilePath=

# Enable TLS with KeyStore type configuration in broker.
useKeyStoreTls=false

# TLS KeyStore type configuration: JKS, PKCS12
tlsTrustStoreType=JKS

# TLS TrustStore path
tlsTrustStorePath=

# TLS TrustStore password
tlsTrustStorePassword=
50 changes: 50 additions & 0 deletions modules/components/attachments/sql/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
extra:
broker: false
brokerSts: true
pulsarSQL: true
proxy: false
wsproxy: false
bastion: false

pulsarSQL:
server:
workers: 1
service:
type: ClusterIP

zookeeper:
replicaCount: 1

bookkeeper:
replicaCount: 1

broker:
component: "broker" # this has to match brokerSts naming due to a bug
replicaCount: 0

brokerSts:
component: "broker" # this has to match broker naming due to a bug
replicaCount: 1
functionsWorkerEnabled: true # to overcome the need for functions working
ledger: # the default assumes 2 bookies but this chart only deploys 1
defaultEnsembleSize: 1
defaultAckQuorum: 1
defaultWriteQuorum: 1

image:
brokerSts:
repository: datastax/lunastreaming-all #because we want connectors too
pullPolicy: IfNotPresent
tag: 2.10_2.4
zookeeper:
repository: datastax/lunastreaming
pullPolicy: IfNotPresent
tag: 2.10_2.4
bookkeeper:
repository: datastax/lunastreaming
pullPolicy: IfNotPresent
tag: 2.10_2.4
pulsarSQL:
repository: datastax/lunastreaming
tag: 2.10_2.4
pullPolicy: IfNotPresent
2 changes: 1 addition & 1 deletion modules/components/pages/admin-console-tutorial.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ The *Admin Console for {pulsar-reg}* is a web-based UI from {company} that admin

In the *{pulsar-short} Admin Console*, you can use {pulsar-short} clients to send and receive pub/sub messages.

If you installed the Admin console with the xref:install-upgrade:quickstart-helm-installs.adoc[{company} Helm chart], access the Admin console via the `pulsar-adminconsole` external load balancer endpoint in your cloud provider:
If you installed the Admin console with the xref:install-upgrade:quickstart-helm-installs.adoc[{company} {pulsar-short} Helm chart], access the Admin console via the `pulsar-adminconsole` external load balancer endpoint in your cloud provider:

image::GCP-all-pods.png[GCP Pods]

Expand Down
2 changes: 1 addition & 1 deletion modules/components/pages/heartbeat-vm.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
= Heartbeat on VM/Server

This document describes how to install {pulsar-short} Heartbeat on a virtual machine (VM) or server. For installation with the Docker image, see xref:install-upgrade:quickstart-helm-installs.adoc[Helm Chart Installation].
This document describes how to install {pulsar-short} Heartbeat on a virtual machine (VM) or server. For installation with the Docker image, see xref:install-upgrade:quickstart-helm-installs.adoc[].

== Install Heartbeat Binary

Expand Down
8 changes: 4 additions & 4 deletions modules/components/pages/pulsar-beam.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
= {pulsar-beam} with {product}
:navtitle: {pulsar-beam}
:description: Install a minimal {product} Helm chart that includes {pulsar-beam}
:helmValuesPath: https://raw.githubusercontent.com/datastaxdevs/luna-streaming-examples/main/beam/values.yaml
:description: Install a minimal {company} {pulsar-short} Helm chart that includes {pulsar-beam}
:helmValuesPath: beam

The {pulsar-beam-repo}[{pulsar-beam}] project is an HTTP-based streaming and queueing system for use with {pulsar}.

Expand All @@ -12,15 +12,15 @@ With {pulsar-beam}, you can send messages over HTTP, push messages to a webhook
include::ROOT:partial$luna-name.adoc[]
====

In this guide, you'll install a minimal {product} Helm chart that includes {pulsar-beam}.
In this guide, you'll install a minimal {company} {pulsar-short} Helm chart that includes {pulsar-beam}.

== Prerequisites

* https://helm.sh/docs/intro/install/[Helm 3 CLI] (this example uses version 3.8.0)
* Enough access to a K8s cluster to create a namespace, deployments, and pods
* https://kubernetes.io/docs/tasks/tools/[Kubectl CLI] (this example uses version 1.23.4)

== Install {product} Helm chart
== Install the {company} {pulsar-short} Helm chart

include::partial$install-helm.adoc[]

Expand Down
22 changes: 8 additions & 14 deletions modules/components/pages/pulsar-sql.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
= Using {pulsar-short} SQL with {product}
:navtitle: {pulsar-short} SQL
:description: This guide installs the {product} Helm chart using minimum values for a working {pulsar-short} cluster that includes SQL workers
:helmValuesPath: https://raw.githubusercontent.com/datastaxdevs/luna-streaming-examples/main/pulsar-sql/values.yaml
:description: This guide installs the {company} {pulsar-short} Helm chart using minimum values for a working {pulsar-short} cluster that includes SQL workers
:helmValuesPath: sql

{pulsar-short} SQL allows enterprises to query {pulsar} topic data with SQL.
This is a powerful feature for an Enterprise, and SQL is a language they're likely familiar with.
Expand All @@ -14,7 +14,7 @@ Additionally, {pulsar-short} has built-in options to create Trino workers and au
include::ROOT:partial$luna-name.adoc[]
====

In this guide, we will use the {product} Helm Chart to install a {pulsar-short} cluster with {pulsar-short} SQL.
In this guide, we will use the {company} {pulsar-short} Helm Chart to install a {pulsar-short} cluster with {pulsar-short} SQL.
The Trino coordinator and desired number of workers will be created directly in the cluster.

== Prerequisites
Expand All @@ -31,7 +31,7 @@ PrestoDB has been replaced by Trino, and {pulsar} is using Trino's version.
The Trino CLI uses the "X-TRINO-USER" header for authentications.
====

== Install {product} Helm chart
== Install the {company} {pulsar-short} Helm chart

include::partial$install-helm.adoc[]

Expand Down Expand Up @@ -76,20 +76,14 @@ The minimalist Helm chart values use the https://github.com/datastax/release-not
This example uses the "public" tenant and "default" namespace.
These are created by default in {pulsar-short}, but you can use whatever tenant and namespace you are comfortable with.

. Download the minimalist {pulsar-short} client.
This "client.conf" assumes the port forwarding addresses we will perform in the next step.
+
[source,shell]
----
wget https://raw.githubusercontent.com/datastaxdevs/luna-streaming-examples/main/client.conf
----
. xref:components:attachment$client.conf[Download the minimalist {pulsar-short} client].
This `client.conf` assumes the port forwarding addresses we will perform in the next step.

. Set the client environment variable. Replace the value with the absolute path to the "client.conf" you just downloaded.
. Set a client configuration environment variable with the absolute path to the downloaded `client.conf` file:
+
[source,shell]
----
# the client conf path must be an absolute path to the downloaded conf file
export PULSAR_CLIENT_CONF=<REPLACE_WITH /path/to/client.conf>
export PULSAR_CLIENT_CONF=/path/to/client.conf
----

. Navigate to the {pulsar-short} home folder and run the following command.
Expand Down
17 changes: 11 additions & 6 deletions modules/components/partials/install-helm.adoc
Original file line number Diff line number Diff line change
@@ -1,21 +1,26 @@
. Add the {company} Helm chart repo to your Helm store:
[IMPORTANT]
====
The {pulsar-helm-chart-repo}[{company} {pulsar-short} Helm chart] is deprecated.
For Kubernetes deployments, use xref:kaap-operator::index.adoc[{kaap}] instead.
====

. Add the {company} {pulsar-short} Helm chart repo to your Helm store:
+
[source,shell]
----
helm repo add datastax-pulsar https://datastax.github.io/pulsar-helm-chart
----

. Install the Helm chart using a minimalist values file.
This command creates a Helm release named "my-pulsar-cluster" using the {company} Luna Helm chart, within the K8s namespace "datastax-pulsar".
This command creates a Helm release named "my-pulsar-cluster" using the {company} {pulsar-short} Helm chart, within the K8s namespace "datastax-pulsar".
The minimal cluster creates only the essential components and has no ingress or load balanced services.
+
[source,shell,subs="attributes+"]
[source,shell]
----
VALUES_URL="{helmValuesPath}"
helm install \
--namespace datastax-pulsar \
--create-namespace \
--values $VALUES_URL \
--values /path/to/values.yaml \
--version 3.0.4 \
my-pulsar-cluster \
datastax-pulsar/pulsar
Expand All @@ -24,7 +29,7 @@ helm install \
. Wait for the broker pod to be in a running state.
You might see a few restarts as your components start up.
+
[source,shell,subs="attributes+"]
[source,shell]
----
kubectl -n datastax-pulsar wait --for=condition=Ready pod/pulsar-broker-0 --timeout=120s
----
4 changes: 2 additions & 2 deletions modules/install-upgrade/pages/production-cluster-sizing.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Scale bookies up on disc usage percentage. Scale down manually by making a booki
[#recommended]
=== Recommended server components

The {product} Helm chart deployment includes optional but highly recommended server components for better {pulsar-short} cluster metrics monitoring and operation visibility.
The {company} {pulsar-short} Helm chart deployment includes optional but highly recommended server components for better {pulsar-short} cluster metrics monitoring and operation visibility.

* https://bookkeeper.apache.org/docs/admin/autorecovery[{bookkeeper-short} AutoRecovery] - This is a {pulsar-short} component that recovers {bookkeeper-short} data in the event of a bookie outage. While optional you will want the insurance of AutoRecovery working on your behalf.
A single instance of AutoRecovery should be adequate - only in the most heavily-used clusters will you need more.
Expand All @@ -59,7 +59,7 @@ Function worker spec is usually focused on compute and memory.
Scale the workers based on overall usage (both CPU and memory).
* xref:luna-streaming:components:admin-console-tutorial.adoc[{pulsar-short} AdminConsole] - This is an optional web-based admin console for managing {pulsar-short} clusters, and makes management much easier than tons of CLI commands. The sizing and scaling for AdminConsole has nothing to do with the cluster, as it is not a failure point.
* xref:luna-streaming:components:heartbeat-vm.adoc[{pulsar-short} Heartbeat] - This is an optional component that monitors the health of {pulsar-short} cluster and emits metrics about the cluster that are helpful for observing and debugging issues.
* Prometheus/Grafana/Alert manager stack - This is the default observability stack for a cluster. The Luna Helm chart includes pre-made dashboards in Grafana and pre-wires all the metrics scraping.
* Prometheus/Grafana/Alert manager stack - This is the default observability stack for a cluster. The {company} {pulsar-short} Helm chart includes pre-made dashboards in Grafana and pre-wires all the metrics scraping.

image::pulsar-components.png[]

Expand Down
Loading