ExamGecko
Search Search Login
Home Home / Google / Professional Data Engineer
Question list
Search
Search

List of questions

Search
Open VPLUS File
Convert VPLUS to PDF

Related questions

You are designing a data mesh on Google Cloud by using Dataplex to manage data in BigQuery and Cloud Storage. You want to simplify data asset permissions. You are creating a customer virtual lake with two user groups: * Data engineers, which require lull data lake access * Analytic users, which require access to curated data You need to assign access rights to these two groups. What should you do?
What are all of the BigQuery operations that Google charges for?
Your infrastructure team has set up an interconnect link between Google Cloud and the on-premises network. You are designing a high-throughput streaming pipeline to ingest data in streaming from an Apache Kafka cluster hosted on-premises. You want to store the data in BigQuery, with as minima! latency as possible. What should you do?
You are designing the architecture to process your data from Cloud Storage to BigQuery by using Dataflow. The network team provided you with the Shared VPC network and subnetwork to be used by your pipelines. You need to enable the deployment of the pipeline on the Shared VPC network. What should you do?
Your company built a TensorFlow neural-network model with a large number of neurons and layers. The model fits well for the training dat a. However, when tested against new data, it performs poorly. What method can you employ to address this?
You need to connect multiple applications with dynamic public IP addresses to a Cloud SQL instance. You configured users with strong passwords and enforced the SSL connection to your Cloud SOL instance. You want to use Cloud SQL public IP and ensure that you have secured connections. What should you do?
You have a BigQuery table that ingests data directly from a Pub/Sub subscription. The ingested data is encrypted with a Google-managed encryption key. You need to meet a new organization policy that requires you to use keys from a centralized Cloud Key Management Service (Cloud KMS) project to encrypt data at rest. What should you do?
You are using BigQuery with a regional dataset that includes a table with the daily sales volumes. This table is updated multiple times per day. You need to protect your sales table in case of regional failures with a recovery point objective (RPO) of less than 24 hours, while keeping costs to a minimum. What should you do?
An external customer provides you with a daily dump of data from their database. The data flows into Google Cloud Storage GCS as comma-separated values (CSV) files. You want to analyze this data in Google BigQuery, but the data could have rows that are formatted incorrectly or corrupted. How should you build this pipeline?
You designed a database for patient records as a pilot project to cover a few hundred patients in three clinics. Your design used a single database table to represent all patients and their visits, and you used self-joins to generate reports. The server resource utilization was at 50%. Since then, the scope of the project has expanded. The database must now store 100 times more patient records. You can no longer run the reports, because they either take too long or they encounter errors with insufficient compute resources. How should you adjust the database design?

Question 163 - Professional Data Engineer discussion

Report
Export

You've migrated a Hadoop job from an on-prem cluster to dataproc and GCS. Your Spark job is a complicated analytical workload that consists of many shuffing operations and initial data are parquet files (on average 200-400 MB size each). You see some degradation in performance after the migration to Dataproc, so you'd like to optimize for it. You need to keep in mind that your organization is very cost-sensitive, so you'd like to continue using Dataproc on preemptibles (with 2 non-preemptible workers only) for this workload.

What should you do?

A.
Increase the size of your parquet files to ensure them to be 1 GB minimum.
Answers
A.
Increase the size of your parquet files to ensure them to be 1 GB minimum.
B.
Switch to TFRecords formats (appr. 200MB per file) instead of parquet files.
Answers
B.
Switch to TFRecords formats (appr. 200MB per file) instead of parquet files.
C.
Switch from HDDs to SSDs, copy initial data from GCS to HDFS, run the Spark job and copy results back to GCS.
Answers
C.
Switch from HDDs to SSDs, copy initial data from GCS to HDFS, run the Spark job and copy results back to GCS.
D.
Switch from HDDs to SSDs, override the preemptible VMs configuration to increase the boot disk size.
Answers
D.
Switch from HDDs to SSDs, override the preemptible VMs configuration to increase the boot disk size.
Suggested answer: C

Explanation:

To optimize the performance of a complex Spark job on Dataproc that heavily relies on shufflingoperations, and given the cost constraints of using preemptible VMs, switching from HDDs toSSDs and using HDFS as an intermediate storage layer can significantly improve performance.Here's why option C is the best choice:Performance of SSDs:SSDs provide much faster read and write speeds compared to HDDs, which is crucial forperformance-intensive operations like shuffling in Spark jobs.Using SSDs can reduce I/O bottlenecks during the shuffle phase of your Spark job, improvingoverall job performance.Intermediate Storage with HDFS:Copying data from Google Cloud Storage (GCS) to HDFS for intermediate storage can reducelatency compared to reading directly from GCS.HDFS provides better locality and faster data access within the Dataproc cluster, which cansignificantly improve the efficiency of shuffling and other I/O operations.

Show Answer
asked 18/09/2024
Maija Janite
38 questions
PreviousPreviousNextNext
User
Your answer:
0 comments
Sorted by

Leave a comment first

ExamGecko

Copyright @2023 ExamGecko | All right reserved

Mail us: [email protected]
About us
Contact us
Twitter X
Facebook
Medium
Convert VPLUS to PDF
Open VPLUS files Easily and Quickly
Privacy Policy
Disclaimer
Terms