[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.16″ global_colors_info=”{}”][et_pb_row admin_label=”row” _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||”][et_pb_text admin_label=”Problem Statement” module_id=”new1″ _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n

Problem Statement<\/h3>\n

At around end of March, I want to get my hands on the old raspberry pi cluster again as I need a testbed for K8S, ChatOps, CI\/CD etc. The DevOps ecosystem in 2023 is more ARM ready compared with 2020 which makes building a usable K8S stack on Pi realistic. I upgraded from a 4 nodes cluster to a 7 Pi4 nodes with POE capabilities, SSD, USB and sitting inside a nice 1U rack. Then spending the next two months’ time on testing various OS. Re-installing the whole stack multiple times and struggling with the home router is fun. At the end the cluster is up with all platform engineering tools deployed.
\n[\/et_pb_text][et_pb_image src=”https:\/\/devwp.visibleone.io\/p225-newweb\/wp-content\/uploads\/2023\/08\/1684592208426-1.jpeg” title_text=”1684592208426″ _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_image][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]From the software perspective it all works fine. However, I quickly realize that the ventilation inside the rack is not good as what I expected. Although each Pi POE head comes with a fan and heatsink added the CPU temperature can easily goes above 60 degrees after an hour or so. Although the Pi rack comes with some nice LED displays with information on the temp of the Pi but that’s a bit small to look at. I need to find a solution to monitor the temperature of the whole stack easily. Ideally the monitoring can be presented on a nice dashboard with alerting capabilities. I decided to work on a solution to address the problem but keep coding to a minimum.
\n[\/et_pb_text][et_pb_text admin_label=”Solution” module_id=”new2″ _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n

Solution<\/h3>\n

I want to share how I derived a solution for this without getting into too much (but unavoidable) technical detail. The goal is to build something fast and easily accessible. At the same time avoid toil in deployment.
\n[\/et_pb_text][et_pb_text admin_label=”Step 1:” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]Step 1: Writing shell script to temperature as metrics in Prometheus scrap-able format.<\/strong><\/em>
\n[\/et_pb_text][et_pb_code _builder_version=”4.21.0″ _module_preset=”default” hover_enabled=”0″ global_colors_info=”{}” module_class=”block-code” sticky_enabled=”0″]#!\/bin\/s while true; do ncat -l -p 7027 –sh-exec ‘\\ C=”rpitemp $(cat \/sys\/class\/thermal\/thermal_zone0\/temp | sed “s\/\\([0-9]\\{2\\}\\)\/\\1.\/”)”; \\ printf “HTTP\/1.0 200 OK\\nContent-Length: ${#C}\\n\\n$C”‘done[\/et_pb_code][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]I am using dietpi OS and this few lines of code can provide me a simple line of “rpitemp 39.5″ from port 7027. Ready for scraping by Prometheus.
\n[\/et_pb_text][et_pb_text admin_label=”Step 2:” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]Step 2: Use ansible to deploy the script to all nodes.<\/strong><\/em>
\n[\/et_pb_text][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]I turned the above into a script, use ansible to copy the file to all nodes and set the script to run at boot. Below is the ansible playbook.yml.
\n[\/et_pb_text][et_pb_code _builder_version=”4.21.0″ _module_preset=”default” hover_enabled=”0″ global_colors_info=”{}” module_class=”block-code” sticky_enabled=”0″]- name: Set cron job for getting temperature from all worker node\u00a0 hosts: workers\u00a0 tasks:\u00a0 – name: Copy temp.sh file to all worker nodes\u00a0 \u00a0 become: true\u00a0\u00a0 \u00a0 copy:\u00a0 \u00a0 \u00a0 src: \/home\/dietpi\/temp.sh\u00a0 \u00a0 \u00a0 dest: \/home\/dietpi\/temp.sh\u00a0 \u00a0 \u00a0 owner: root\u00a0 \u00a0 \u00a0 group: root\u00a0 \u00a0 \u00a0 \u00a0\u00a0\u00a0 \u00a0 \u00a0 mode: 0700\u00a0 – name: Create an entry in crontab for getting the pi temp on every restart\u00a0 \u00a0 ansible.builtin.cron:\u00a0 \u00a0 \u00a0 name: “a job for reboot”\u00a0 \u00a0 \u00a0 special_time: reboot\u00a0 \u00a0 \u00a0 job: “\/home\/dietpi\/temp.sh &”[\/et_pb_code][et_pb_text admin_label=”Step 3:” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]Step 3: Start a prometheus docker on the control node and scrap temperature metrics from all worker nodes.<\/strong><\/em>
\n[\/et_pb_text][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]Below is the prometheus.yml file. Using remote_write to send metrics from all worker nodes to Grafana Cloud.
\n[\/et_pb_text][et_pb_code _builder_version=”4.21.0″ _module_preset=”default” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]global:\u00a0 scrape_interval: 15s \u00a0 evaluation_interval: 15s scrape_configs:\u00a0 – job_name: “piet_temp_control01”\u00a0 \u00a0 metrics_path: ‘\/’\u00a0 \u00a0 static_configs:\u00a0 \u00a0 \u00a0 – targets: [“10.0.0.100:7027”]\u00a0 \u00a0 \u00a0 – targets: [‘10.0.0.101:7027’]\u00a0 \u00a0 \u00a0 – targets: [‘10.0.0.102:7027’]\u00a0 \u00a0 \u00a0 – targets: [‘10.0.0.103:7027’]\u00a0 \u00a0 \u00a0 – targets: [‘10.0.0.104:7027’]\u00a0 \u00a0 \u00a0 – targets: [‘10.0.0.105:7027’]\u00a0 \u00a0 \u00a0 – targets: [‘10.0.0.106:7027’]# remote write locationremote_write:\u00a0 – url: [granfana cloud URL]\u00a0 \u00a0 basic_auth:\u00a0 \u00a0 \u00a0 username: [username]\u00a0 \u00a0 \u00a0 password: [password][\/et_pb_code][et_pb_text admin_label=”Step 4:” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]Step 4: Sign up a Grafana cloud free tier account<\/strong><\/em>[\/et_pb_text][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]The free tier account comes with 10 free dashboards quota, that’s enough for my simple use case. I choose to use Grafana cloud for building the dashboard to avoid the trouble of getting into the Pi cluster to access the local Grafana instance. Also, it will be easier for alert integration using Grafana Cloud.
\n[\/et_pb_text][et_pb_text admin_label=”Step 5:” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]Step 5: Building the dashboard<\/strong><\/em>
\n[\/et_pb_text][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]This is probably the easiest part. Grafana comes with nice dash-boarding capability for different kind of use cases. I haven’t finished setting up the alert as I am still struggling how I want to get the alert. Looking at the figures of the dashboard I probably need to open the case for better ventilation or add additional 40mm fans to the bottom or back of the rack for better cooling. Hopefully I can run this cluster 24×7 after solving the temperature issue.
\n[\/et_pb_text][et_pb_image src=”https:\/\/devwp.visibleone.io\/p225-newweb\/wp-content\/uploads\/2023\/08\/1684591631611.png” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_image][et_pb_text _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]The free tier account comes with 10 free dashboards quota, that’s enough for my simple use case. I choose to use Grafana cloud for building the dashboard to avoid the trouble of getting into the Pi cluster to access the local Grafana instance. Also, it will be easier for alert integration using Grafana Cloud.
\n[\/et_pb_text][et_pb_text admin_label=”Conclusion” module_id=”new3″ _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n

Conclusion<\/h3>\n

This is an interesting side project outside of the K8S cluster on DietPi OS. Raspberry pi’s are common today as SBC for edge computing and remote IoT use cases for digital displays and edge gateways. Remember temperature is another important metric to monitor for your edge computing devices to ensure system availability and reliability.
\n[\/et_pb_text][et_pb_team_member name=”Lindsay Chung” position=”Head of Solutions Engineering” linkedin_url=”https:\/\/www.linkedin.com\/in\/lindsay-chung-813a0429\/” _builder_version=”4.21.0″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_team_member][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"

At around end of March, I want to get my hands on the old raspberry pi cluster again as I need a testbed for K8S, ChatOps, CI\/CD etc. The DevOps ecosystem in 2023 is more ARM ready compared with 2020 which makes building a usable K8S stack on Pi realistic. I upgraded from a 4 nodes cluster to a 7 Pi4 nodes with POE capabilities, SSD, USB and sitting inside a nice 1U rack. Then spending the next two months’ time on testing various OS. Re-installing the whole stack multiple times and struggling with the home router is fun. At the end the cluster is up with all platform engineering tools deployed.<\/p>\n","protected":false},"author":1,"featured_media":10732,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","inline_featured_image":false,"footnotes":""},"categories":[122],"tags":[],"class_list":["post-9485","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog-cn"],"acf":[],"yoast_head":"\n